Practical Algorithms for Incremental Software Development Environments

ion: Clients should be able to view the program (or any subset of it) with or without whitespace, and both viewsshould be equally efficient to produce. Flexibility: The representation should be independent of the language, the whitespace model, the number and types ofwhitespace categories, and the mechanisms by which they are specified and discovered during lexical anal-ysis. The implementation is straightforward: each contiguous sequence of whitespace tokens is represented as a balancedbinary tree. (Often this ‘tree’ will consist of a single whitespace token.) The sequence is logically associated with the pre-ceding token that represents an instance of a terminal symbol in the grammar. An additional connector node is introducedto serve as the parent of the both the preceding terminal and the root of the whitespace sequence. Figure C.1 illustratesthe arrangement. Whitespace at the beginning of the program follows the bos sentinel token (Figure 3.2), maintaininguniformity in the representation.3This approach meets all the criteria described above. Ordering is obviously maintained; the original program text canbe reconstituted simply by walking the tree, inspecting the content of each lexeme encountered. Clients can navigate white-space sequences with the same structural operations used elsewhere in the program, and whitespace tokens are implementedin the same fashion as other nodes. 3The use of out-of-context parsing for error recovery requires the ability to re-attach and merge whitespace sequences, which is supported effectivelyby the representation described here. Section 8.5.2 discusses the interaction between out-of-context analysis and the presence of extra-grammatical tokensin more detail. C.2. INTEGRATING WHITESPACE WITH THE PROGRAM STRUCTURE127